feat(lidar_centerpoint): output the covariance of pose and twist #6573

tzhong518 · 2024-03-08T05:39:57Z

Description

This PR enables the centerpoint node to update the covariance of covariance of pose and twist.
The variance comes from the model trained with sigma loss(https://github.com/tier4/AWMLdetection3D/pull/133), which predicts the variance of the boundingbox's size, position, and velocity.

Tests performed

ros2 launch lidar_centerpoint lidar_centerpoint.launch.xml model_name:=centerpoint_sigma input/pointcloud:=/sensing/lidar/concatenated/pointcloud
output:
`ros2 topic echo ~/objects

existence_probability: 0.47262266278266907
classification:
- label: 2
  probability: 1.0
  kinematics:
  pose_with_covariance:
  pose:
  position:
  x: 14.863276481628418
  y: -30.522504806518555
  z: 0.982421875
  orientation:
  x: 0.0
  y: -0.0
  z: -0.8441231237133663
  w: -0.5361493747197593
  covariance:
  - 0.2047557532787323
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.23383840918540955
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.1902950257062912
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.025285644456744194
    has_position_covariance: true
    orientation_availability: 1
    twist_with_covariance:
    twist:
    linear:
    x: 6.219586886509506e-05
    y: 0.0
    z: 0.0
    angular:
    x: 0.0
    y: 0.0
    z: 7.58850622177124
    covariance:
  - 0.06898762285709381
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.09374400973320007
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
  - 0.0
    has_twist: true
    has_twist_covariance: true
    shape:
    type: 0
    footprint:
    points: []
    dimensions:
    x: 5.908353805541992
    y: 2.103604555130005
    z: 2.882314920425415
    ---`

Notes for reviewers

Interface changes

Effects on system behavior

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

I've confirmed the contribution guidelines.
The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

The PR follows the pull request guidelines.
The PR has been properly tested.
The PR has been reviewed by the code owners.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

There are no open discussions or they are tracked via tickets.
The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

kminoda

Thank you for the PR 👍

Please fix the conflict before opening the PR, and also some CIs those are failing 🙏

FYI: https://tier4.atlassian.net/wiki/spaces/~727523079/pages/2417199892/Pull+Request

codecov · 2024-04-03T02:24:46Z

Codecov Report

Attention: Patch coverage is 0% with 32 lines in your changes are missing coverage. Please review.

Project coverage is 14.93%. Comparing base (35642a6) to head (8a23731).
Report is 27 commits behind head on main.

❗ Current head 8a23731 differs from pull request most recent head e595630. Consider uploading reports for the commit e595630 to get more accurate results

Files	Patch %	Lines
perception/lidar_centerpoint/lib/ros_utils.cpp	0.00%	18 Missing ⚠️
...t/include/lidar_centerpoint/centerpoint_config.hpp	0.00%	7 Missing ⚠️
perception/lidar_centerpoint/src/node.cpp	0.00%	4 Missing ⚠️
...ion_based_fusion/src/pointpainting_fusion/node.cpp	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6573      +/-   ##
==========================================
- Coverage   14.93%   14.93%   -0.01%     
==========================================
  Files        1944     1939       -5     
  Lines      134061   133700     -361     
  Branches    39891    39756     -135     
==========================================
- Hits        20027    19971      -56     
+ Misses      91735    91483     -252     
+ Partials    22299    22246      -53

Flag	Coverage Δ		*Carryforward flag
differential	`0.00% <0.00%> (?)`
total	`14.94% <ø> (+<0.01%)`	⬆️	Carriedforward from 44ded7a

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: tzhong518 <[email protected]>

tzhong518 · 2024-04-03T03:05:22Z

Config of pointpainting is updated.

github-actions · 2024-04-03T03:06:06Z

Possible Configurations:

pr_reviewer.require_score_review = False
pr_reviewer.require_tests_review = True
pr_reviewer.require_estimate_effort_to_review = True
pr_reviewer.require_can_be_split_review = False
pr_reviewer.require_soc2_ticket = False
pr_reviewer.soc2_ticket_prompt = 'Does the PR description include a link to ticket in a project management system (e.g., Jira, Asana, Trello, etc.) ?'
pr_reviewer.num_code_suggestions = 4
pr_reviewer.inline_code_comments = False
pr_reviewer.ask_and_reflect = False
pr_reviewer.persistent_comment = True
pr_reviewer.extra_instructions = ''
pr_reviewer.final_update_message = True
pr_reviewer.enable_review_labels_security = True
pr_reviewer.enable_review_labels_effort = True
pr_reviewer.require_all_thresholds_for_incremental_review = False
pr_reviewer.minimal_commits_for_incremental_review = 0
pr_reviewer.minimal_minutes_for_incremental_review = 0
pr_reviewer.enable_help_text = True
pr_reviewer.enable_auto_approval = False
pr_reviewer.maximal_review_effort = 5

pr_description.publish_labels = True
pr_description.add_original_user_description = True
pr_description.keep_original_user_title = True
pr_description.use_bullet_points = True
pr_description.extra_instructions = ''
pr_description.enable_pr_type = True
pr_description.final_update_message = True
pr_description.enable_help_text = False
pr_description.enable_help_comment = True
pr_description.publish_description_as_comment = False
pr_description.publish_description_as_comment_persistent = True
pr_description.enable_semantic_files_types = True
pr_description.collapsible_file_list = 'adaptive'
pr_description.inline_file_summary = False
pr_description.use_description_markers = False
pr_description.include_generated_by_header = True

pr_questions.enable_help_text = True

pr_code_suggestions.max_context_tokens = 8000
pr_code_suggestions.num_code_suggestions = 4
pr_code_suggestions.summarize = True
pr_code_suggestions.extra_instructions = ''
pr_code_suggestions.rank_suggestions = False
pr_code_suggestions.enable_help_text = True
pr_code_suggestions.persistent_comment = False
pr_code_suggestions.auto_extended_mode = True
pr_code_suggestions.num_code_suggestions_per_chunk = 5
pr_code_suggestions.max_number_of_calls = 3
pr_code_suggestions.parallel_calls = True
pr_code_suggestions.rank_extended_suggestions = False
pr_code_suggestions.final_clip_factor = 0.8

pr_add_docs.extra_instructions = ''
pr_add_docs.docs_style = 'Sphinx Style'

pr_update_changelog.push_changelog_changes = False
pr_update_changelog.extra_instructions = ''

pr_test.extra_instructions = ''
pr_test.testing_framework = ''
pr_test.num_tests = 3
pr_test.avoid_mocks = True
pr_test.file = ''
pr_test.class_name = ''
pr_test.enable_help_text = True

pr_improve_component.num_code_suggestions = 4
pr_improve_component.extra_instructions = ''
pr_improve_component.file = ''
pr_improve_component.class_name = ''

pr_similar_issue.skip_comments = False
pr_similar_issue.force_update_dataset = False
pr_similar_issue.max_issues_to_scan = 500
pr_similar_issue.vectordb = 'pinecone'

technolojin · 2024-04-03T05:42:07Z

I am testing the centerpoint_sigma model with lsim

### copy onnx models
# $HOME/autoware_data/lidar_centerpoint/pts_backbone_neck_head_centerpoint_sigma.onnx
# $HOME/autoware_data/lidar_centerpoint/pts_voxel_encoder_centerpoint_sigma.onnx

### launch once for tensorRT cuda engine generation: convert onnx to engine
ros2 launch lidar_centerpoint lidar_centerpoint.launch.xml model_name:=centerpoint_sigma input/pointcloud:=/sensing/lidar/concatenated/pointcloud

### copy parameter to autoware_launch
# autoware/launcher/autoware_launch/config/perception/object_recognition/detection/lidar_model/centerpoint_sigma.param.yaml

### launch lsim
ros2 launch autoware_launch logging_simulator.launch.xml  ........ lidar_detection_model:=centerpoint centerpoint_model_name:=centerpoint_sigma

centerpoint_sigma2.mp4

technolojin · 2024-04-03T07:56:01Z

It would be better to add available option list in dnn detector launcher
https://github.com/autowarefoundation/autoware.universe/blob/250f0818f8694bb94d4c5613602f9c52996b6848/launch/tier4_perception_launch/launch/object_recognition/detection/detector/lidar_dnn_detector.launch.xml#L8C14-L8C36

<arg name="centerpoint_model_name" default="centerpoint_sigma" description="options: `centerpoint`, `centerpoint_tiny`, `centerpoint_sigma`"/>

YoshiRi · 2024-04-03T09:59:31Z

Compatibility check

I checked this PR would not brake conventional detection process with sample-rosbag in autoware tutorial.

config
model: conventional one
autoware_launch is not changed
result
Not broken. Covariance and twist is not published.

Twist check

Since current model already has twist output, we can get twist by changing paramter in autoware_launch.
feat(lidar_centerpoint, multi_object_tracker) : Use velocity estimated by CenterPoint to stabilize initial velocity of tracker #6602

Pose covariance check

Current model seems not able to measure covariance, but the process still works.

config
- add variance=true in conventional model param

YoshiRi

It worked! But I think we need to fix autoware_launch first.
Could you make PR and link it to this?

perception/image_projection_based_fusion/src/pointpainting_fusion/node.cpp

perception/lidar_centerpoint/src/node.cpp

Co-authored-by: Yoshi Ri <[email protected]>

…ion/node.cpp Co-authored-by: Yoshi Ri <[email protected]>

Co-authored-by: Yoshi Ri <[email protected]>

Signed-off-by: tzhong518 <[email protected]>

tzhong518 · 2024-04-04T01:54:56Z

PR to autoware_launch: autowarefoundation/autoware_launch#945

kminoda

I left one minor comment.

Also, let ius discuss in slack whether to handle this as a centerpoint_model or different model (e.g. lidar_model = centerpoint_sigma)

perception/lidar_centerpoint/lib/postprocess/postprocess_kernel.cu

Signed-off-by: tzhong518 <[email protected]>

YoshiRi

LGTM

technolojin

Can you fix the launcher misconfiguration?

...perception_launch/launch/object_recognition/detection/detector/lidar_dnn_detector.launch.xml

…tion/detector/lidar_dnn_detector.launch.xml Co-authored-by: Taekjin LEE <[email protected]>

technolojin

LGTM

yukkysaito

LGTM

…owarefoundation#6573) * feat: postprocess variance Signed-off-by: tzhong518 <[email protected]> * feat: output variance Signed-off-by: tzhong518 <[email protected]> * feat: add has_variance to config Signed-off-by: tzhong518 <[email protected]> * fix: single_inference node Signed-off-by: tzhong518 <[email protected]> * style(pre-commit): autofix * fix: add to pointpainting param Signed-off-by: tzhong518 <[email protected]> * Update perception/lidar_centerpoint/src/node.cpp Co-authored-by: Yoshi Ri <[email protected]> * Update perception/image_projection_based_fusion/src/pointpainting_fusion/node.cpp Co-authored-by: Yoshi Ri <[email protected]> * Update perception/lidar_centerpoint/src/node.cpp Co-authored-by: Yoshi Ri <[email protected]> * fix: add options Signed-off-by: tzhong518 <[email protected]> * fix: avoid powf Signed-off-by: tzhong518 <[email protected]> * Update launch/tier4_perception_launch/launch/object_recognition/detection/detector/lidar_dnn_detector.launch.xml Co-authored-by: Taekjin LEE <[email protected]> --------- Signed-off-by: tzhong518 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Yoshi Ri <[email protected]> Co-authored-by: Taekjin LEE <[email protected]>

github-actions bot added the component:perception Advanced sensor data processing and environment understanding. (auto-assigned) label Mar 8, 2024

tzhong518 marked this pull request as ready for review March 14, 2024 01:54

tzhong518 requested review from knzo25 and kminoda as code owners March 14, 2024 01:54

kminoda reviewed Mar 14, 2024

View reviewed changes

kminoda added the run:build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) label Apr 3, 2024

fix: add to pointpainting param

a7708a4

Signed-off-by: tzhong518 <[email protected]>

YoshiRi requested changes Apr 3, 2024

View reviewed changes

perception/image_projection_based_fusion/src/pointpainting_fusion/node.cpp Outdated Show resolved Hide resolved

perception/lidar_centerpoint/src/node.cpp Outdated Show resolved Hide resolved

perception/lidar_centerpoint/src/node.cpp Outdated Show resolved Hide resolved

tzhong518 and others added 4 commits April 4, 2024 10:10

Update perception/lidar_centerpoint/src/node.cpp

29c655d

Co-authored-by: Yoshi Ri <[email protected]>

Update perception/image_projection_based_fusion/src/pointpainting_fus…

375c023

…ion/node.cpp Co-authored-by: Yoshi Ri <[email protected]>

Update perception/lidar_centerpoint/src/node.cpp

4ff31ff

Co-authored-by: Yoshi Ri <[email protected]>

fix: add options

44ded7a

Signed-off-by: tzhong518 <[email protected]>

github-actions bot added the component:launch Launch files, scripts and initialization tools. (auto-assigned) label Apr 4, 2024

tzhong518 mentioned this pull request Apr 4, 2024

feat(autoware_launch): add centerpoint_sigma param autowarefoundation/autoware_launch#945

Merged

7 tasks

kminoda reviewed Apr 4, 2024

View reviewed changes

perception/lidar_centerpoint/lib/postprocess/postprocess_kernel.cu Outdated Show resolved Hide resolved

fix: avoid powf

8a23731

Signed-off-by: tzhong518 <[email protected]>

YoshiRi approved these changes Apr 9, 2024

View reviewed changes

technolojin requested changes Apr 12, 2024

View reviewed changes

...perception_launch/launch/object_recognition/detection/detector/lidar_dnn_detector.launch.xml Outdated Show resolved Hide resolved

technolojin assigned tzhong518 Apr 12, 2024

Update launch/tier4_perception_launch/launch/object_recognition/detec…

62274ea

…tion/detector/lidar_dnn_detector.launch.xml Co-authored-by: Taekjin LEE <[email protected]>

technolojin approved these changes Apr 12, 2024

View reviewed changes

Merge branch 'main' into feat/sigma_loss

e595630

tzhong518 enabled auto-merge (squash) April 16, 2024 01:08

yukkysaito approved these changes Apr 16, 2024

View reviewed changes

kminoda approved these changes Apr 16, 2024

View reviewed changes

tzhong518 merged commit cf5a2ed into autowarefoundation:main Apr 16, 2024
21 of 22 checks passed

tzhong518 mentioned this pull request Apr 18, 2024

feat(autoware_launch): add centerpoint_sigma param to pointpainting.param.yaml autowarefoundation/autoware_launch#955

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(lidar_centerpoint): output the covariance of pose and twist #6573

feat(lidar_centerpoint): output the covariance of pose and twist #6573

tzhong518 commented Mar 8, 2024 •

edited

Loading

kminoda left a comment •

edited

Loading

codecov bot commented Apr 3, 2024 •

edited

Loading

tzhong518 commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

technolojin commented Apr 3, 2024 •

edited

Loading

technolojin commented Apr 3, 2024

YoshiRi commented Apr 3, 2024

YoshiRi left a comment

tzhong518 commented Apr 4, 2024

kminoda left a comment

YoshiRi left a comment

technolojin left a comment

technolojin left a comment

yukkysaito left a comment

feat(lidar_centerpoint): output the covariance of pose and twist #6573

feat(lidar_centerpoint): output the covariance of pose and twist #6573

Conversation

tzhong518 commented Mar 8, 2024 • edited Loading

Description

Related links

Tests performed

Notes for reviewers

Interface changes

Effects on system behavior

Pre-review checklist for the PR author

In-review checklist for the PR reviewers

Post-review checklist for the PR author

kminoda left a comment • edited Loading

Choose a reason for hiding this comment

codecov bot commented Apr 3, 2024 • edited Loading

Codecov Report

tzhong518 commented Apr 3, 2024

github-actions bot commented Apr 3, 2024

technolojin commented Apr 3, 2024 • edited Loading

technolojin commented Apr 3, 2024

YoshiRi commented Apr 3, 2024

Compatibility check

Twist check

Pose covariance check

YoshiRi left a comment

Choose a reason for hiding this comment

tzhong518 commented Apr 4, 2024

kminoda left a comment

Choose a reason for hiding this comment

YoshiRi left a comment

Choose a reason for hiding this comment

technolojin left a comment

Choose a reason for hiding this comment

technolojin left a comment

Choose a reason for hiding this comment

yukkysaito left a comment

Choose a reason for hiding this comment

tzhong518 commented Mar 8, 2024 •

edited

Loading

kminoda left a comment •

edited

Loading

codecov bot commented Apr 3, 2024 •

edited

Loading

technolojin commented Apr 3, 2024 •

edited

Loading